Interactive Reinforcement Learning Research Articles

Advancing Household Robotics: Deep Interactive Reinforcement Learning for Efficient Training and Enhanced Performance

The market for domestic robots—made to perform household chore, is growing as these robots relieve people of everyday responsibilities. Domestic robots are generally welcomed for their role in easing human labour, in contrast to industrial robots, which are frequently criticised for displacing human workers. But before these robots can carry out domestic chores, they need to become proficient in a number of minor activities, such as recognizing their surroundings, making decisions, and picking up on human behaviours. Reinforcement learning, or RL, has emerged as a key robotics technology that enables robots to interact with their environment and learn how to optimize their actions in order to maximize rewards. However, the goal of Deep Reinforcement Learning (DeepRL) is to address more complicated, continuous action-state spaces in real-world settings by combining RL with Neural Networks (NNs). The efficacy of DeepRL can be further augmented through interactive feedback, in which a trainer offers real-time guidance to expedite the robot's learning process. Nevertheless, the current methods have drawbacks, namely the transient application of guidance that results in repeated learning under identical conditions. Therefore, we present a novel method to preserve and reuse information and advice via Deep Interactive Reinforcement Learning (DeepIRL)—it utilizes a persistent rule-based system. This method not only expedites the training process but also lessens the number of repetitions that instructors will have to carry out. This study has the potential to advance the development of household robots and improve their effectiveness and efficiency as learners.

Towards interactive reinforcement learning with intrinsic feedback

Reinforcement learning (RL) and brain-computer interfaces (BCI) have experienced significant growth over the past decade. With rising interest in human-in-the-loop (HITL), incorporating human input with RL algorithms has given rise to the sub-field of interactive RL. Adjacently, the field of BCI has long been interested in extracting informative brain signals from neural activity for use in human–computer interactions. A key link between these fields lies in the interpretation of neural activity as feedback such that interactive RL approaches can be employed. We denote this new and emerging medium of feedback as intrinsic feedback. Despite intrinsic feedback’s ability to be conveyed automatically and even unconsciously, proper exploration surrounding this key link has largely gone unaddressed by both communities. Thus, to help facilitate a deeper understanding and a more effective utilization, we provide a tutorial-style review covering the motivations, approaches, and open problems of intrinsic feedback and its foundational concepts.

Intelligent Robotics in Pediatric Cooperative Neurorehabilitation: A Review

The landscape of neurorehabilitation is undergoing a profound transformation with the integration of artificial intelligence (AI)-driven robotics. This review addresses the pressing need for advancements in pediatric neurorehabilitation and underscores the pivotal role of AI-driven robotics in addressing existing gaps. By leveraging AI technologies, robotic systems can transcend the limitations of preprogrammed guidelines and adapt to individual patient needs, thereby fostering patient-centric care. This review explores recent strides in social and diagnostic robotics, physical therapy, assistive robotics, smart interfaces, and cognitive training within the context of pediatric neurorehabilitation. Furthermore, it examines the impact of emerging AI techniques, including artificial emotional intelligence, interactive reinforcement learning, and natural language processing, on enhancing cooperative neurorehabilitation outcomes. Importantly, the review underscores the imperative of responsible AI deployment and emphasizes the significance of unbiased, explainable, and interpretable models in fostering adaptability and effectiveness in pediatric neurorehabilitation settings. In conclusion, this review provides a comprehensive overview of the evolving landscape of AI-driven robotics in pediatric neurorehabilitation and offers valuable insights for clinicians, researchers, and policymakers.

An efficient reinforcement learning scheme for the confinement escape problem

Crucial real-world problems in robotics like trajectory planning during convoy missions and autonomous rescue missions can be framed as a confinement escape problem (CEP) (a type of pursuit-evasion game). In a typical CEP, an evader attempts to escape a confinement region by sequentially making decisions to plan an escape while the region is patrolled by multiple smart pursuers. The evader has a limited sensing range and does not know the total number of pursuers and their pursuit strategies making it difficult to model the environment and obtain a generalizable escape strategy. In this paper, the CEP is formulated in a reinforcement learning (RL) framework to overcome the above difficulties. The state function is designed independent of the total number of pursuers and the shape of the confinement region thereby making the RL approach scalable. To handle training consistency issues in deep RL and convergence issues due to sparse rewards, a Scaffolding Reflection based Reinforcement Learning (SR2L) approach is presented in this paper where the SR2L employs an actor–critic method with a motion planner scaffold to accelerate its training speed. Performance evaluation of SR2L shows that it trains twice as fast compared to other existing state-of-the-art actor–critic RL methods. Performance results show that the convergence of SR2L is more consistent than the corresponding conventional actor–critic RL methods and interactive reinforcement learning methods. Monte-Carlo simulation results show that SR2L outperforms other conventional RL methods and the motion planner with at least 28% and 10% faster escape times respectively while having the lowest variance in escape times against different pursuit strategies. Ablation studies done by changing different environmental parameters clearly show the scalability and generalizability of the proposed SR2L approach.

Deep reinforcement learning for cooperative robots based on adaptive sentiment feedback

Human–robot cooperative tasks have gained importance with the emergence of robotics and artificial intelligence technology. In interactive reinforcement learning techniques, robots learn target tasks by receiving feedback from an experienced human trainer. However, most interactive reinforcement learning studies require a separate process to integrate the trainer’s feedback into the training dataset, making it challenging for robots to learn new tasks from humans in real-time. Furthermore, the types of feedback sentences that trainers can use are limited in previous research. To address these limitations, this paper proposes a robot teaching strategy that uses deep RL via human–robot interaction to learn table balancing tasks interactively. The proposed system employs Deep Q-Network with real-time sentiment feedback delivered through the trainer’s speech to learn cooperative tasks. We designed a novel reward function that incorporates sentiment feedback from human speech in real-time during the learning process. The paper presents an improved reward shaping technique based on subdivided feedback levels and shrinking feedback. This function serves as a guide for the robot to engage in natural interactions with humans and enables it to learn the tasks effectively. Experimental results demonstrate that the proposed interactive deep reinforcement learning model achieved a high success rate of up to 99.06%, outperforming the model without sentiment feedback.

Interactive reinforcement learning innovation to reduce carbon emissions in railway infrastructure maintenance

Carbon emission is one of the primary contributors to global warming. The global community is paying great attention to this negative impact. The goal of this study is to reduce the negative impact of railway maintenance by applying reinforcement learning (RL) by optimizing maintenance activities. Railway maintenance is a complex process that may not be efficient in terms of environmental aspect. This study is the world's first to use the potential of RL to reduce carbon emission from railway maintenance. The data used to create the RL model are gathered from the field data between 2016–019. The study section is 30 km long. Proximal Policy Optimization (PPO) is applied in the study to develop the RL model. The results demonstrate that using RL reduces carbon emission from railway maintenance by 48%, which generates a considerable amount of carbon emission reduction and reduces railway defects by 68%, which also improves maintenance efficiency significantly.

Online POI Recommendation: Learning Dynamic Geo-Human Interactions in Streams

Online POI recommendation strives to recommend future visitable places to users over time. It is crucial to improve the user experience of location-based social networking applications (e.g., Google Maps, Yelp). While numerous studies focus on capturing user visit preferences, geo-human interactions are ignored. However, users make visit decisions based on the status of geospatial contexts. In the meantime, such visits will change the status of geospatial contexts. Therefore, disregarding such geo-human interactions in streams will result in inferior recommendation performance. To fill this gap, in this paper, we propose a novel deep interactive reinforcement learning framework to model geo-human interactions. Specifically, this framework has two main parts: the representation module and the imitation module. The purpose of the representation module is to capture geo-human interactions and convert them into embedding vectors (state). The imitation module is a reinforced agent whose job is to imitate user visit behavior by recommending next-visit POI (action) based on the state. Imitation performance is regarded as a reward signal to optimize the whole interactive framework. When the model converges, the imitation module can precisely perceive users and geospatial contexts to provide accurate POI recommendations. Finally, we conduct extensive experiments to validate the superiority of our framework.

Towards a Broad-Persistent Advising Approach for Deep Interactive Reinforcement Learning in Robotic Environments.

Deep Reinforcement Learning (DeepRL) methods have been widely used in robotics to learn about the environment and acquire behaviours autonomously. Deep Interactive Reinforcement 2 Learning (DeepIRL) includes interactive feedback from an external trainer or expert giving advice to help learners choose actions to speed up the learning process. However, current research has been limited to interactions that offer actionable advice to only the current state of the agent. Additionally, the information is discarded by the agent after a single use, which causes a duplicate process at the same state for a revisit. In this paper, we present Broad-Persistent Advising (BPA), an approach that retains and reuses the processed information. It not only helps trainers give more general advice relevant to similar states instead of only the current state, but also allows the agent to speed up the learning process. We tested the proposed approach in two continuous robotic scenarios, namely a cart pole balancing task and a simulated robot navigation task. The results demonstrated that the agent's learning speed increased, as evidenced by the rising reward points of up to 37%, while maintaining the number of interactions required for the trainer, in comparison to the DeepIRL approach.

Quantifying the effect of feedback frequency in interactive reinforcement learning for robotic tasks

Reinforcement learning (RL) has become widely adopted in robot control. Despite many successes, one major persisting problem can be very low data efficiency. One solution is interactive feedback, which has been shown to speed up RL considerably. As a result, there is an abundance of different strategies, which are, however, primarily tested on discrete grid-world and small scale optimal control scenarios. In the literature, there is no consensus about which feedback frequency is optimal or at which time the feedback is most beneficial. To resolve these discrepancies we isolate and quantify the effect of feedback frequency in robotic tasks with continuous state and action spaces. The experiments encompass inverse kinematics learning for robotic manipulator arms of different complexity. We show that seemingly contradictory reported phenomena occur at different complexity levels. Furthermore, our results suggest that no single ideal feedback frequency exists. Rather that feedback frequency should be changed as the agent’s proficiency in the task increases.

Interactive Reinforcement Learning With Bayesian Fusion of Multimodal Advice

Interactive Reinforcement Learning With Bayesian Fusion of Multimodal Advice

Human engagement providing evaluative and informative advice for interactive reinforcement learning

Interactive reinforcement learning proposes the use of externally sourced information in order to speed up the learning process. When interacting with a learner agent, humans may provide either evaluative or informative advice. Prior research has focused on the effect of human-sourced advice by including real-time feedback on the interactive reinforcement learning process, specifically aiming to improve the learning speed of the agent, while minimising the time demands on the human. This work focuses on answering which of two approaches, evaluative or informative, is the preferred instructional approach for humans. Moreover, this work presents an experimental setup for a human trial designed to compare the methods people use to deliver advice in terms of human engagement. The results obtained show that users giving informative advice to the learner agents provide more accurate advice, are willing to assist the learner agent for a longer time, and provide more advice per episode. Additionally, self-evaluation from participants using the informative approach has indicated that the agent’s ability to follow the advice is higher, and therefore, they feel their own advice to be of higher accuracy when compared to people providing evaluative advice.

Automated Feature Selection: A Reinforcement Learning Perspective

Feature selection is a critical step in machine learning that selects the most important features for a subsequent prediction task. Effective feature selection can help to reduce dimensionality, improve prediction accuracy, and increase result comprehensibility. It is traditionally challenging to find the optimal feature subset from the feature subset space as the space could be very large. While much effort has been made on feature selection, reinforcement learning can provide a new perspective towards a more globally-optimal searching strategy. In the preliminary work, we propose a multi-agent reinforcement learning framework for the feature selection problem. Specifically, we first reformulate feature selection with a reinforcement learning framework by regarding each feature as an agent. Besides, we obtain the state of the environment in three ways, i.e., statistic description, autoencoder, and graph convolutional network (GCN), in order to derive a fixed-length state representation as the input of reinforcement learning. In addition, we study how the coordination among feature agents can be improved by a more effective reward scheme. Also, we provide a GMM-based generative rectified sampling strategy to accelerate the convergence of multi-agent reinforcement learning. Our method searches the feature subset space more globally and can be easily adapted to real-time scenarios due to the nature of reinforcement learning. In the extended version, we further accelerate the framework from two aspects. From the sampling aspect, we show the indirect acceleration by proposing a rank-based softmax sampling strategy. From the exploration aspect, we show the direct acceleration by proposing an interactive reinforcement learning (IRL)-based exploration strategy. Extensive experimental results show the significant improvement of the proposed method over conventional approaches.

A conceptual framework for externally-influenced agents: an assisted reinforcement learning review

A long-term goal of reinforcement learning agents is to be able to perform tasks in complex real-world scenarios. The use of external information is one way of scaling agents to more complex problems. However, there is a general lack of collaboration or interoperability between different approaches using external information. In this work, while reviewing externally-influenced methods, we propose a conceptual framework and taxonomy for assisted reinforcement learning, aimed at fostering collaboration by classifying and comparing various methods that use external information in the learning process. The proposed taxonomy details the relationship between the external information source and the learner agent, highlighting the process of information decomposition, structure, retention, and how it can be used to influence agent learning. As well as reviewing state-of-the-art methods, we identify current streams of reinforcement learning that use external information in order to improve the agent's performance and its decision-making process. These include heuristic reinforcement learning, interactive reinforcement learning, learning from demonstration, transfer learning, and learning from multiple sources, among others. These streams of reinforcement learning operate with the shared objective of scaffolding the learner agent. Lastly, we discuss further possibilities for future work in the field of assisted reinforcement learning systems.

Persistent rule-based interactive reinforcement learning

Interactive reinforcement learning has allowed speeding up the learning process in autonomous agents by including a human trainer providing extra information to the agent in real-time. Current interactive reinforcement learning research has been limited to real-time interactions that offer relevant user advice to the current state only. Additionally, the information provided by each interaction is not retained and instead discarded by the agent after a single-use. In this work, we propose a persistent rule-based interactive reinforcement learning approach, i.e., a method for retaining and reusing provided knowledge, allowing trainers to give general advice relevant to more than just the current state. Our experimental results show persistent advice substantially improves the performance of the agent while reducing the number of interactions required for the trainer. Moreover, rule-based advice shows similar performance impact as state-based advice, but with a substantially reduced interaction count.

Influencing Reinforcement Learning through Natural Language Guidance

Interactive reinforcement learning (IRL) agents use human feedback or instruction to help them learn in complex environments. Often, this feedback comes in the form of a discrete signal that’s either positive or negative. While informative, this information can be difficult to generalize on its own. In this work, we explore how natural language advice can be used to provide a richer feedback signal to a reinforcement learning agent by extending policy shaping, a well-known IRL technique. Usually policy shaping employs a human feedback policy to help an agent to learn more about how to achieve its goal. In our case, we replace this human feedback policy with policy generated based on natural language advice. We aim to inspect if the generated natural language reasoning provides support to a deep RL agent to decide its actions successfully in any given environment. So, we design our model with three networks: first one is the experience driven, next is the advice generator and third one is the advice driven. While the experience driven RL agent chooses its actions being influenced by the environmental reward, the advice driven neural network with generated feedback by the advice generator for any new state selects its actions to assist the RL agent to better policy shaping.

Reinforcement Learning Approaches in Social Robotics.

This article surveys reinforcement learning approaches in social robotics. Reinforcement learning is a framework for decision-making problems in which an agent interacts through trial-and-error with its environment to discover an optimal behavior. Since interaction is a key component in both reinforcement learning and social robotics, it can be a well-suited approach for real-world interactions with physically embodied social robots. The scope of the paper is focused particularly on studies that include social physical robots and real-world human-robot interactions with users. We present a thorough analysis of reinforcement learning approaches in social robotics. In addition to a survey, we categorize existent reinforcement learning approaches based on the used method and the design of the reward mechanisms. Moreover, since communication capability is a prominent feature of social robots, we discuss and group the papers based on the communication medium used for reward formulation. Considering the importance of designing the reward function, we also provide a categorization of the papers based on the nature of the reward. This categorization includes three major themes: interactive reinforcement learning, intrinsically motivated methods, and task performance-driven methods. The benefits and challenges of reinforcement learning in social robotics, evaluation methods of the papers regarding whether or not they use subjective and algorithmic measures, a discussion in the view of real-world reinforcement learning challenges and proposed solutions, the points that remain to be explored, including the approaches that have thus far received less attention is also given in the paper. Thus, this paper aims to become a starting point for researchers interested in using and applying reinforcement learning methods in this particular research field.

An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users.

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

A Robust Approach for Continuous Interactive Actor-Critic Algorithms

Reinforcement learning refers to a machine learning paradigm in which an agent interacts with the environment to learn how to perform a task. The characteristics of the environment may change over time or be affected by disturbances not controlled, avoiding the agent finding a proper policy. Some approaches attempt to address these problems, as interactive reinforcement learning, where an external entity helps the agent learn through advice. Other approaches, such as robust reinforcement learning, allow the agent to learn the task, acting in a disturbed environment. In this paper, we propose an approach that addresses interactive reinforcement learning problems in a dynamic environment, where advice provides information on the task and the dynamics of the environment. Thus, an agent learns a policy in a disturbed environment while receiving advice. We implement our approach in the dynamic version of the cart-pole balancing task and a simulated robotic arm dynamic environment to organize objects. Our results show that the proposed approach allows an agent to complete the task satisfactorily in a dynamic, continuous state-action domain. Moreover, experimental results suggest agents trained with our approach are less sensitive to changes in the characteristics of the environment than interactive reinforcement learning agents.

Interactive Reinforcement Learning for Feature Selection with Decision Tree in the Loop

We study the problem of balancing effectiveness and efficiency in automated feature selection. After exploring many feature selection methods, we observe a computational dilemma: 1) traditional feature selection is mostly efficient, but difficult to identify the best subset; 2) the emerging reinforced feature selection automatically navigates to the best subset, but is usually inefficient. Can we bridge the gap between effectiveness and efficiency under automation? Motivated by this dilemma, we aim to develop a novel feature space navigation method. In our preliminary work, we leveraged interactive reinforcement learning to accelerate feature selection by external trainer-agent interaction. In this journal version, we propose a novel interactive and closed-loop architecture to simultaneously model interactive reinforcement learning (IRL) and decision tree feedback (DTF). Specifically, IRL is to create an interactive feature selection loop and DTF is to feed structured feature knowledge back to the loop. First, the tree-structured feature hierarchy from decision tree is leveraged to improve state representation. In particular, we represent the selected feature subset as an undirected graph of feature-feature correlations and a directed tree of decision features. We propose a new embedding method capable of empowering graph convolutional network to jointly learn state representation from both the graph and the tree. Second, the tree-structured feature hierarchy is exploited to develop a new reward scheme. In particular, we personalize reward assignment of agents based on decision tree feature importance. In addition, observing agents' actions can be feedback, we devise another reward scheme, to weigh and assign reward based on the feature selected frequency ratio in historical action records. Finally, we present extensive experiments on real-world datasets to show the improved performance.

New interactive agent based reinforcement learning approach towards smart generator bidding in electricity market with micro grid integration

In order to suit the needs of the dynamically changing electricity market, software developers have developed various tools taking in to account the need of artificial intelligence for the electricity market entities. Algorithms in artificial intelligence are often divided into either supervised, unsupervised and reinforcement learning approach. A reinforcement learning when compared to supervised and unsupervised learning makes use of agent to learn from interaction with an environment and receives rewards based on the action it takes. It either exploits or explores in finding a solution. In the deregulated power market, the GenCos are modeled as agent by which the GenCo learns the market environment as agent and explores to get profited The Multi-agent based simulation is an effective method to incorporate this sort of intelligence and for providing efficient communication among the market entities. Using Multi-agent system, the problem existing in electricity market can be reduced since each entity problem can be solved by an individual agent. Multi-agent based reinforcement learning algorithm is used to handle the electricity market data. Here an agent based computational framework named Agent Based Modeling of Electricity Systems (AMES) under Java platform is developed for the design of electricity market. Market Agents balances the supply and demand through Market Clearing Price (no congestion) and Locational Marginal Price (congestion management) by performing optimal power flow. The agent also maximizes the profit of the Generator Companies (GenCo’s) through new learning strategy proposed using Variant Roth–Erev (VRE) interactive reinforcement learning method towards smart bidding among GenCo’s. The congestion relieving action in the transmission line and its effects on GenCo learning is discussed in this paper. The analysis is carried out on the electricity whole sale market functioning on a day-ahead basis developed by means of location and timing of injection of power. IEEE-3 bus system and IEEE-30 bus system with microgrid considered as non-dispatchable load is considered using agent based analysis. This technique helps the GenCo’s to attain possible high net earnings even with microgrid integration thus helps to relieve the congestion in the transmission lines.